Skip to content

Data flow visualiser: dfd@1 model, static and runtime extraction, renderer, CLI and panel route - #8

Draft
Portll wants to merge 46 commits into
fix/red-ci-20261112from
feat/dfd-20261112
Draft

Portll wants to merge 46 commits into
fix/red-ci-20261112from
feat/dfd-20261112

Conversation

@Portll

@Portll Portll commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #7. Retarget to main after #7 lands.

What it adds

  • Model (schema/dfd.schema.json, lib/dfd.mjs): dfd@1 with processes, stores and external entities. Stores take subtypes broker, queue, topic, stream, log, database and file; externals take endpoint, scheduler, actor, system, env and binary. Flows carry mode (sync, async, event), role (data, feedback, error, retry), existence and direction, plus groups and maturity layers.
  • Analysis: two-way pairs, feedback (an error or retry return on an adjacent pair counts as feedback), self-loops, hop ranks, branch and merge points, source-to-sink paths, async hops, and a hop filter that follows direction.
  • Static extraction (flow/static.mjs, flow/triggers.mjs):
    • URL call sites and .listen become endpoints.
    • DatabaseSync(path) plus SQL verbs become database reads and writes. This also fixes SQL passed to db.exec being read as a spawn (28 cases).
    • A spawned node is linked to the script it runs.
    • launchd, workflow schedule, timer and fs.watch become triggers.
    • Append-then-tail pairs are marked.
  • Runtime witness (flow/trace-hook.cjs, flow/reconcile.mjs): network connects and listens, checked by a self-probe so a hook that sees nothing cannot report a clean run. Every open records its access mode, so touches edges get a direction, and bare-filename stores resolve to the paths actually opened.
  • Adapter (flow/dfd-adapter.mjs): fromFlowGraph with reconcile and layer-map options, and fromCodeFlows for a scanned repository's CodeQL flows.
  • Renderer (lib/dfd-render.mjs): Gane–Sarson notation in one self-contained page with a hash-pinned CSP. The layout is deterministic, with collision-free label placement, a transpose step, groups kept together, group collapse and automatic left-to-right or top-to-bottom orientation.
  • CLI (flow/dfd-cli.mjs) and panel route /dfd/ and /api/dfd. flow/ is now in npm files.
  • Authored diagrams: docs/dfd/threat-model.dfd.json and docs/dfd/commitwork-layers.json.
  • Spec: docs/SPEC-dataflow-oversight.md gains section C6.

Verified

Linux, Node 22.23: lib/test/dfd*, flow/test/* and admin/test/dfd* give 663 tests, 0 fail. 77 tree-walking census files and the doc-generator checks pass, except tests that fail here only because this machine runs as root or because of its git version. Line coverage of every new or changed source file is at least 97%. Not run here: macOS, Windows, Node 26, the full suite.

Limits (data, not code)

  • Most database paths and URLs in this tree are computed, so they are reported as unanalysable sites (365) rather than nodes.
  • There are no tail pairs in this tree yet, so no brokers are drawn.
  • fromCodeFlows draws source and sink only, because the CodeQL records keep a step count, not the steps.

dfd@1 (schema/dfd.schema.json): processes, stores (broker, queue, topic, stream
subtypes) and external entities, with groups, maturity layers and sync, async
or event flows. lib/dfd.mjs validates and analyses: two-way pairs, feedback
edges, self-loops, hop ranks, branch and merge points, source-to-sink paths,
async hops. lib/dfd-render.mjs draws Gane-Sarson notation as one self-contained
page with a deterministic layered layout. flow/dfd-cli.mjs renders a dfd@1 file
or the flow graph.
# Conflicts:
#	flow/README.md
# Conflicts:
#	flow/dfd-cli.mjs
…s on disjoint ranks, pull sources to their first consumer, and give returning flows their own edge
…aliser's flags, adapter home and route in flow/README.md
assert.deepEqual([usage, absent, invalid, nomatch, toobig].map((r) => r.status), [400, 404, 422, 422, 422]);
assert.match(absent.body, /flow graph not found at .*flow\.json — run `node flow\/static\.mjs`/);
assert.match(nomatch.body, /prefix "<script>"/);
assert.doesNotMatch(nomatch.body, /<script>/);
Comment thread flow/triggers.mjs

/** Body `{…}` range of a function named `name` declared in this module, or null. */
export function functionBody(masked, name) {
const n = name.replace(/\$/g, '\\$');
Comment thread flow/triggers.mjs
}
const locals = [...fsNamed].filter(([, imported]) => imported === 'watch' || imported === 'watchFile').map(([l]) => l);
if (locals.length) {
const re = new RegExp(`(?<![\\w$.])(${locals.map((l) => l.replace(/\$/g, '\\$')).join('|')})\\s*\\(`, 'g');
const urls = [...html.matchAll(/url\(([^)]*)\)/g)].map((x) => x[1]);
const links = [...html.matchAll(/(?:href|src|xlink:href|action|data|poster|srcset)="([^"]*)"/g)].map((x) => x[1]);
const inAttributes = [...html.matchAll(/\s[\w:-]+="([^"]*https?:\/\/[^"]*)"/g)].map((x) => x[1]);
const script = /<script>([\s\S]*?)<\/script>/.exec(html)[1];
// act
const { html, csp } = renderHtml(m);
const style = /<style>([\s\S]*?)<\/style>/.exec(html)[1];
const script = /<script>([\s\S]*?)<\/script>/.exec(html)[1];

// act
const { html } = renderHtml(big, { maxPaths: 5 });
const fact = (k) => new RegExp(`<dt>${k.replace(/[()]/g, '\\$&')}</dt><dd>([^\\n]*)</dd>`).exec(html)[1];

// assert
assert.match(html, /<tspan x="[\d.]+">monitor\/extractors\/sast-lint\/<\/tspan><tspan x="[\d.]+" dy="11.5">_deno-check-counts-and-more\.mjs<\/tspan>/);
assert.match(html, new RegExp(`<title>P1 process: ${label.replace(/[.]/g, '\\.')}</title>`));
// assert
assert.match(html, /<tspan x="[\d.]+">monitor\/extractors\/sast-lint\/<\/tspan><tspan x="[\d.]+" dy="11.5">_deno-check-counts-and-more\.mjs<\/tspan>/);
assert.match(html, new RegExp(`<title>P1 process: ${label.replace(/[.]/g, '\\.')}</title>`));
assert.match(html, new RegExp(`<td class="m">${label.replace(/[.]/g, '\\.')} ext</td>`));
Comment thread flow/static.mjs
if (!args.length) continue;
const target = resolveValue(ctx, args[0]);
if (!target.ok || target.value !== 'node') continue;
let info = { candidates: [] };
Portll pushed a commit that referenced this pull request Nov 12, 2026
…s holding unread pipe data

On a refusal the helper SIGKILLs bwrap. If bwrap had not yet read the
resolv.conf written to fd 5, the helper's socketpair end failed with
ECONNRESET, which nothing handled: the helper crashed with exit 1 and a
stack, and a caller read the lane as failed instead of not run. This is the
intermittent bin/test/sandbox-netns-live.test.mjs:138 failure (1 !== 125)
seen on PRs #5 and #8. Pipe errors after a kill are now expected; any other
pipe error is a refusal. bin/test/sandbox-net-refusal-pipes.test.mjs
reproduces it with a fake bwrap that never reads fd 5: 0/15 before, 15/15
after.
Portll added a commit that referenced this pull request Nov 12, 2026
…s holding unread pipe data

On a refusal the helper SIGKILLs bwrap. If bwrap had not yet read the
resolv.conf written to fd 5, the helper's socketpair end failed with
ECONNRESET, which nothing handled: the helper crashed with exit 1 and a
stack, and a caller read the lane as failed instead of not run. This is the
intermittent bin/test/sandbox-netns-live.test.mjs:138 failure (1 !== 125)
seen on PRs #5 and #8. Pipe errors after a kill are now expected; any other
pipe error is a refusal. bin/test/sandbox-net-refusal-pipes.test.mjs
reproduces it with a fake bwrap that never reads fd 5: 0/15 before, 15/15
after.
@Portll
Portll force-pushed the fix/red-ci-20261112 branch from 6a4bc7c to 019f5c9 Compare November 12, 2026 03:56
The static pass takes any git checkout: its tracked .mjs .cjs .js .jsx .ts .tsx .mts .cts files,
TypeScript through node:module stripTypeScriptTypes in strip mode (offsets kept), CommonJS
parsed under the module wrapper with vm.Script. The launchd generator is imported only for this
checkout; elsewhere it is a site, not read. Unreadable, too-large and unsupported-language files
and other languages are counted in the summary.
…clean

--repo runs the static pass over another checkout (re-running itself with
--experimental-vm-modules when needed); --codeql reads SARIF or dataflow records; both
together merge by repository-relative path. source.unknown counts what was not read and
the page banner states it.
Dispatched ahead of first-run setup, needs no report directory, listed in usage() and the
unknown-command hint. exit-codes reads a delegated module's header outside bin/, monitor/
and cra/. npm-pack test asserts the command's import closure ships.
Only a name the panel's fleet resolver knows; a path, traversal or unknown name is 400 with
the reason. The pass runs in a child; its model goes through the same prepare().
A comment quoting a dynamic import('./x') named a module nobody imports.
The CommonJS refusal alone ('import outside a module') hid the real one: decorators and
using declarations, which V8 on Node 22 does not parse.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants